Large-scale standardized tests are generally limited to certain cognitive processes and behaviors that they require of examinees. Multiple-choice questions have been favored because they can be scored objectively, quickly and at low cost. Scores derived from multiple-choice questions typically have good predictive and concurrent validity. However, such questions generally only test for recognition of correct answers and do not require constructed responses which more appropriately conform to the kinds of behavior that teachers want their students to express. More generally, standardized test questions are perceived as failing to elicit the full range of desirable cognitive processes, which for many educators include thinking critically, synthesizing ideas, and formulating and carrying out plans.
Multiple-choice questions, or items, are frequently the target of the above criticism, in part because they are commonly thought to require no more than recall of information. Moreover, in answering multiple-choice questions, examinees might capitalize on abilities that have little to do with the construct being measured. For example, researchers have identified a response elimination strategy whereby examinees eliminate implausible distractors, and then guess from the remaining options. Unless the target construct is intended to embrace such strategizing, error is introduced because the items discriminate on the basis of abilities that lie beyond the pale of the construct. For these and other reasons for dissatisfaction with multiple-choice testing, there has been an effort to try to develop alternatives. The products of these efforts have included performance and portfolio-based assessments. As discussed in the above cross-referenced application Ser. No. 382,716, incorporated herein by reference, an additional alternative is the use of figural response items. Figural response testing differs from multiple-choice in at least two ways. First, figural response items call for constructed responses, i.e., answers made up in the mind of the examinee rather than chosen from a list of options. A second feature of figural response items is their dependence upon figural material, such as illustrations and graphs, as the response medium. The examinee responds to questions or directions by carrying out operations on a figure, or on a background containing figural representations. The cross-referenced patent provides illustrations of figural response items where the examinee forms the task by manually making one or more marks in relation to the background figural media, i.e., graphs, illustrations, and diagrams.
Our investigations have shown that figural response tasks, or items, are found to be somewhat more difficult and also more reliable (predictive of overall ability) than multiple-choice items. It is expected that figural response testing will reduce or eliminate the utility of strategies as are known to be employed in multiple-choice testing, thereby enhancing the estimates of the target proficiency. Constructed response items have also been shown to be particularly useful for diagnosis, since examinees are unconstrained and uncued in the expression in what they know. Moreover, the use of computers makes diagnostic branching possible, so the response to one question might lead to a follow-up question to pinpoint or confirm an area of competence or weakness. Our work has progressed beyond the concepts set forth in the above-referenced patent, to provide for unique advantages and capabilities through a computerized figural response testing system and method. By the techniques that we have invented, we have met a number of needs. The computerized system and method of this invention provide greatly increased flexibility both in presentation of an item, and in method of response. This system also provides greatly increased reliability, e.g., less ambiguity in deciphering responses. It also provides a great improvement in automatic scoring, including the option of on-line scoring, i.e., providing the examinee with a score directly after completion of an item.